Document Clustering using Small World Communities

Previous research has shown the words in natural language documents exist as a small world network. Thus it might be feasible to use extensive physics algorithms for extracting community structure. We present a novel method for semantically clustering a large collection of documents using small world communities. We combine specially modified physics algorithms with traditional information retrieval techniques. A term network is generated from the document collection, the terms are clustered into small world communities, and the semantic term clusters are used to generate overlapping document clusters. Clustering 90K documents took 20 seconds, generating good quality community clusters in nearly linear running time, O(n log n) where n is the size of the lexicon in the document collection.